In FIG. 1 there is illustrated a typical computer system 20 having a host computer 21 with a processor 22 and associated memory 23, one or more data storage subsystems 24, 25 each having a plurality of hard disk drives 26, 27, first and second storage controllers 28, 29 coupled between the host computer and the storage subsystem by interfaces and communications links of conventional type, such as for example, busses or network connections, When the first and second controllers 28, 29 are caching storage controllers (described below), each controller includes a cache memory 30, 31 that serves as intermediate storage. Usually the cache memory is fast random access memory (RAM) of which there are many types.
Increasingly, there is a need to provide access to stored information or data on hard disk drives (or other storage devices) from a plurality of host servers and to also permit the data stored on any particular storage device to be accessed through alternative device controllers. Providing access to the data from multiple hosts eliminates the need to store the data at more than one location (though the data may still be redundantly stored using known mirroring or Redundant Array of Independent Disk (RAID) techniques) and in theory assures that the identical data can be accessed by interested parties. Providing accesses to a storage device through multiple controllers, provides redundant access to the device from an alternate (or second) controller so that the data remains accessible in the event that the first controller fails.
Although providing access to storage devices through multiple controllers is desirable, such a configuration may present data consistency problems. Data consistency refers to all controllers providing visibility to one identical copy of the data. Data consistency can be provided through data synchronization or data coherency or both. Data coherency refers to maintaining a consistent copy of the data in each of the controllers caches. Data synchronization refers to keeping the data in the storage controller's cache the same as that in the storage device.
A storage controller is a device which is capable of directing and data traffic from the host system to one or more non-volatile storage devices. It may or may not have an intermediary cache to stage data between the non-volatile storage device and the host system. A caching controller (or caching storage controller) is a device which is capable of directing the data traffic from a host system to one or more non-volatile storage devices which uses an intermediary data storage device (the cache memory) to stage data between the non-volatile storage device and the host system. In general, the intermediary storage device is built out of RAM to allow a quicker access time to the data. Furthermore, it provides a buffer in which to allow exclusive-or (XOR) operations to be completed for RAID 5 operations. Multiple active controllers are defined as a collection of storage controllers or caching storage controllers which work in a cooperative manner with each other. They provide the ability for recovering from a controller failure by allowing multiple paths to a storage volume. The storage volume is a contiguous range of randomly accessible sector of data. For practical purposes, the sector numbering starts at 0 and goes to N, where N is the total number of sectors available to the host system. A data extent is a range of data within a storage volume delineated by a starting sector and an ending sector. The storage volume is broken up into a number of data extents which are not required to be of the equivalent sizes, but may not overlap. These concepts are used in the discussion of the background and the detailed description of embodiments of the invention, and apply to both.
Caching storage controllers that work independently of one another to store information or data to a secondary storage unit, such as a hard disk drive, or tape unit, are conventionally available. There are also caching storage controllers that work with one or more other controller(s) to provide multiple controller access to a secondary storage unit and provide a fault tolerant environment. If two controllers are simultaneously providing access to a common set of storage devices and each is able to take over the other's functionality in the event of a failure, then those controllers are referred to as active-active or dual-active controllers.
Computer system configurations involving one of more host computers and having two or more controllers that use cache technologies, with access to a storage device through any of the controllers, should desirably provide some mechanism of ensuring that the cache data in each controller is always correct. Unfortunately in conventional systems they may not. Controllers using the SCSI command set could use two commands that are provided in that command set, the "Reserve LUN" and "Release LUN" commands, when LUN is an abbreviation for Logical Unit Number. (SCSI commands, including the Reserve LUN and Release LUN commands, are described in standard references including SCSI-2 Small Computer System Interface-2 ANSI X3.131:1994: which is incorporated herein by reference.) The host computer, especially one operating in a cluster environment, could use these two commands to reserve all accesses to the entire LUN.
Unfortunately, not all host computers use these SCSI commands. Furthermore, the Reserve LUN and Release LUN commands do not provide for reservation of a portion of a storage volume because they apply to the entire storage volume.
The following description is provided relative to FIG. 2, and sets forth the problems with data synchronization between controllers which maintain local cached copies of data. This example details one set of host transactions that could cause data integrity problems (data consistency and data synchronization problems). The data consistency problem is brought about by the fact that each controller's cache operates independently.
With reference to the illustration in FIG. 2, there is shown a portion of computer system 31, including host computer 32 having a processor or central processing unit (CPU) 33, first controller (controller "A") 34, second controller (controller "B") 35, a storage subsystem 36 including at least one backing storage volume 37. Each of the two controllers 34, 35 further separately include first cache (Cache "A") 44 and second cache (Cache "B") 45 for caching data retrieved from backing storage volume 37. Generic techniques for controller caching are known in the art and not described further here. (The inventive structure and method provide certain features, including inventive data structures and methods, that are different from conventional caches and solve the problems associated with conventional controllers and caches, and these differences are described in greater detail in connection with the detailed description of embodiments of the invention.)
Backing storage volume 37 is coupled to each of first and second controllers 34, 35 by storage interface channels 38, 39, and host computer or processor (CPU) 33 is coupled to the controllers by CPU-to-storage device interface 40. The interface 40 may typically be implemented as a Personal Computer Interface (PCI), parallel SCSI, fibre channel, or IEEE-1394 (fire-wire) interface using a storage, file system, or other communications protocol. In like manner, the controller-to-storage device interfaces 38, 39 may typically be implemented using the same set of interfaces and protocols as just described for interface 40. A logical unit number (LUN) is assigned or otherwise associated with each backing storage volume 37. The relationship between physical devices or portions thereof and logical devices is known in the art and not further described here.
In this configuration, if data is written to a logical unit, such as backing storage volume 37A through first controller 34, the data is properly retained in the first controller's cache, that is within cache 44. If data is subsequently written to logical storage unit 37A through second controller 35, the newly written data in backing storage volume 37A matches the data in the second controller's cache 45, but the information in the first controller's cache 44 will not have been updated, and (if the newly written data is different from the original data) no longer matches the data written to the backing storage volume 37A. If a request to read the data is made through first controller 34, the data will be read from cache 44 according to standard data caching and retrieval practices to minimize backing storage volume access, and the wrong information will be returned to the requestor. The data is said to lack coherency between different locations (that is between one or more of the backing storage volume, cache 44, and cache 45), and is out of temporal synchronization as a result of the time-order of events involving the read, write, and caching operations.
Stating the problem in somewhat more concrete terms, in a system with two controllers 34, 35 attached to the same CPU/storage interface and sharing access to a backing storage volume 37, as illustrated in FIG. 2, host computer 33 writes data pattern "AAAA" to the backing storage volume through first controller 34. First controller 34 retains this information in its data cache 44 so that future requests for the data can be fulfilled without having to access the disk backing storage volume 37A. At a later time, the host computer writes the data pattern "BBBB" to backing storage volume 37A at the same location the "AAAA" data pattern had been stored, but now the write operation is directed through the second controller 35 as illustrated in FIG. 3. First controller 34 still has the "AAAA" data pattern stored in its cache 44, but second controller 35 has the "BBBB" data pattern stored in its cache 45. The data in cache 44 ("AAAA") and the data in cache 45 ("BBBB"), each supposedly representing the identical data, no longer match and are incoherent.
The correct data pattern on backing storage volume 37A ("BBBB") is the later data pattern also stored in cache 45 ("BBBB"), but if the host computer 33 attempts to read the information from backing storage volume 37A through first controller 34, first controller 34 will, using conventional techniques, be unaware of any controller 35 operations, and in particular will be unaware that a write operation has altered the data on the backing storage volume. Lacking knowledge that the data has changed, first controller 34 will access it's own cache 44 to retrieve the data, and erroneously return that data pattern ("AAAA") rather than the correct data pattern ("BBBB") to the requesting host computer 33.
Absent some technique for coordinating the actions of the two controllers 34, 35 and their caches 44, 45, or of making one controller aware of the activities of the other controller, data integrity using conventional methods cannot be assured. Therefore, there remains a need for apparatus and method to maintain data integrity under these circumstances.